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DETAILED ACTION 
Response to Arguments 

1. Applicant's arguments on page 8, first paragraph, filed September 7, 2004, with respect 
to claims 1, 2 and 13, have been fully considered and are persuasive. The objections to claims 1, 
2 and 13 have been withdrawn. 

2. Applicant's arguments on pages 6-13, filed September 7, 2004, with respect to the 
rejection(s)of claim(s) 1-12 andl3 under 35 U.S.C. 103(a) and 35 U.S.C. 102(e), respectively, 
have been fully considered and are persuasive. Therefore, the rejections have been withdrawn. 
However, upon further consideration, a new ground(s) of rejection is made in view of Spight 
(USPN 4,462,046, newly cited). 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 1, 3, 4 and 8-13 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Spight (USPN 4,462,046, newly cited) in view of Corby, Jr. et al. (USPN 5,745,387, newly cited, 
hereafter Corby). 

5. Regarding claims 1 and 8, Spight discloses a teaching model generating method and 
device for image processing, in which a subject object has the same or substantially similar shape 
as that of a reference object (column 1, line 7-12), the device comprising: an image processing 
system with which current three-dimensional orientation of the subject object relative to an 
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image pickup device is recognized based on a plurality of predetermined teaching models (stored 
data of desired object) of the reference object (column 3, line 26-47; column 8, line 10-15); and 
an image-capture system (video detector 40, Figure 1; column 5, line 42-48), in advance of the 
recognizing, generating and storing the plurality of teaching models (reference signals) on the 
basis of respective image data produced by taking images of said reference object from a 
plurality of directions, so that the image data respectively obtained at each of said different 
image pickup positions, is stored as a teaching model (reference signal) (column 9, line 1-19). 
Spight does not disclose that one of the reference object and said image pickup device is fixed to 
a movable and positionable part of the robot or is grasped with a hand of the robot, that said 
robot is operated for positioning to a plurality of different image pickup positions and directions, 
or that direction information indicating the respective different direction is stored with the image 
data as a teaching model 

6. Corby discloses an augmented reality maintenance system employing a manipulator arm 
with an archive and comparison device wherein a distal end 100 of the manipulator arm 10 is 
attached to a utility package 1 1 , which may include a spatial imaging device such as a video 
camera (Figure 1; column 4, line 18-33). Corby also discloses a position and attitude sensing 
unit 21 to determine the position and orientation distal end 10b, which can then be used by 
manipulator arm renderer 33 to create several images of a prestored model of the manipulator 
arm from model storage 47 from several different viewpoints and environment renderer 35 to 
produce a number of images of the environment corresponding to supplied viewpoints (column 
4, line 34-65). Corby goes on to disclose an archive and comparison device 50 that utilizes a 
sensor data storage device 51 capable of storing spatial imagery with location, orientation and 
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acquisition parameters linked to each image (column 6, line 29-37). It would have been obvious 
to one of ordinary skill in the art at the time the invention was made to fix an image pickup 
device to a movable and positionable part of a robot, operate the robot for positioning to a 
plurality of different image pickup positions and directions, and store direction information 
indicating the respective different direction with the image data as a teaching model as taught by 
Corby in order to define the identity of the site imaged, when it was imaged, the viewpoint, the 
modality of the imager and description of values relating to the image (column 6, line 37-51). 

7. Note also that Spight does not explicitly disclose that the recognized current orientation 
of the subject is three-dimensional (determine object's position and orientation in space ; column 

8, line 10-15) and Corby only mentions it in passing (column 2, line 6-12; column 8, line 41-45). 
The examiner takes Official Notice that determining three-dimensional orientation of an object is 
well known in the art of robotic vision systems. It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to determine the three-dimensional orientation 
of an object in order to determine the orientation at which to pick up an object. 

8. Regarding claim 3, Spight discloses that said teaching model is a part of the image data 
of the reference object (column 9, line 10-19; column 10, line 50-63). 

9. Regarding claim 4, Spight discloses that said teaching model comprises data obtained by 
performing image processing on the image data of the reference object (column 9, line 10-19; 
column 10, line 50-63). 

10. Regarding claims 9 and 12 (see above discussion of claim 1), Spight discloses a method 
of automatic orientation recognition, comprising: generating and storing a set of images of 
different relative orientations (arrangements; claim 12) of a subject (column 9, line 1-19), the 
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images having been captured by a plurality of robotic operations corresponding to the different 
relative orientations (arrangements) of the subject (Corby; column 4, line 43-65), and associating 
with each image information indicating its respective relative orientation (arrangement) of the 
subject (Corby column 6, line 29-51); after the generating and storing, from a known current 
orientation (arrangement) of (an image pickup device on) a robot, capturing a current image of a 
workpiece that has an unknown orientation (arrangement) relative to an image pickup device on 
the robot (Corby; column 4, line 18-33) (before the robot has come into contact with the 
workpiece; claim 9 only), where the workpiece has a shape substantially similar to the shape of 
the subject (Spight; column 3, line 29-38; column 6, line 1 1-13); after the capturing, using 
pattern matching (correlation) to match one of the stored images with the current image (column 
7, line 28-column 8, line 15; column 9, line 59-column 10, line 1 1); and after the pattern 
matching, (and before the robot has come into contact with the workpiece; claim 9 only), 
determining the orientation (arrangement) of the workpiece relative to the image pickup device 
on the robot based on the relative orientation (arrangement) information associated with the 
matched stored image (reference signal), and also based on the known current orientation 
(arrangement) of the robot (column 8, line 16-37; column 11, line 21-38). 

1 1 . Regarding claim 10, Spight discloses automatically maneuvering the robot to the 
workpiece based at least on the determined orientation of the workpiece relative to the robot 
(column 8, line 16-37). 

12. Regarding claim 11, Spight discloses that the generating and storing is performed for a 
plurality of differently shaped subjects (store a plurality of configurations of each desired object; 
column 9, line 1-19), wherein the current image includes a plurality of differently shaped 
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workpieces, and wherein the pattern matching further comprises identifying workpieces from 
among the plurality of differently shaped workpieces using the images and orientation 
information of the plurality of differently shaped subjects (column 6, line 11-13; column 7, line 
50-column 8, line 15; column 11, line 29-38). 

13. Regarding claim 13 (see above discussion of claim 1), Spight discloses a method 
comprising: robotically taking images of a subject with different three-dimensional subject- 
camera arrangements that vary in three dimensions (column 9, line 1-19; Corby, column 4, line 
17-33), and associating with each image or data thereof information indicating its subject-camera 
arrangement (Corby, column 6, line 29-51); then taking a current image of a workpiece shaped 
like the subject (Spight; column 3, line 29-38; column 6, line 1 1-13); and then before picking up 
the workpiece determining a current workpiece-camera orientation by matching (correlating) one 
of the images or data thereof with the current image (Spight, column 7, line 28-column 8, line 
15; column 9, line 59-column 10, line 1 1), and using predetermined subject-camera arrangement 
information of the matched image to determine the three-dimensional orientation of the 
workpiece relative to the camera (Spight, column 8, line 16-37; column 11, line 21-38). 

14. Claims 2, 5 and 6 are rejected under 35 U.S.C. 103(a) as being unpatentable over Spight 
in view of Corby in view of Werth et al. (USPN 4,504,970, previously cited, hereafter Werth). 

1 5. Regarding claim 2, Spight discloses a teaching model generating device for image 
processing, in which a subject object has same or substantially similar shape as that of a 
reference object, the device comprising: an image processing system with which a current three- 
dimensional orientation of the subject object relative to an image pickup device is recognized by 
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carrying out pattern matching processing of an image of the subject based on a plurality of pre- 
determined teaching models of the reference object; and an image capture system, in advance of 
the recognizing, generating and storing the plurality of teaching models on the basis of respective 
image data produced by taking images of said reference object from a plurality of directions, 
wherein said image pickup device is fixed to a movable and positionable part of a robot or is 
grasped with a hand of the robot, which is operated for positioning to a plurality of different 
relative image pickup positions and directions, so that the image data respectively obtained at 
each of said different image pickup positions is stored as a teaching model (see above discussion 
of claim 1). 

16. Neither Spight nor Corby discloses that the reference object is fixed to a movable part of 
a first robot or is grasped with a hand of the first robot. Werth discloses a training controller for 
pattern processing system wherein it is suggested that an application could utilize two robot 
arms, one which holds a camera which visually guides it to observe a precise assembly point and 
a second which brings a tool or assembly within the visual field of the camera where it is visually 
guided through an operation (column 5, line 12-17). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to utilize two robot arms, one to hold 
a camera and one to hold a tool or assembly as taught by Werth in order to provide more degrees 
of freedom allowing more views of the workpiece from different directions and to provide proper 
alignment for mating parts in automated assembly operations or move a tool to a specific point 
on the part (column 5, line 6-11). 

17. Regarding claim 5, Spight discloses that said teaching model is generated for every 
direction in which said image pickup device took the image of said reference object (column 9, 
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line 1-19) and Corby discloses that said teaching model is stored in association with the 
information on the direction (column 6, line 29-41). 

18. Regarding claim 6, Spight discloses that said image pickup device 40 is a camera 
(column 5, line 42-48). 

19. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Spight in view of 
Corby in view of Werth as applied to claim 2 above, and further in view of Ninomiya et al. 
(USPN 4,61 1,292, previously cited, hereafter Ninomiya). 

20. Spight, Corby nor Werth disclose that said image pickup device is a three-dimensional 
visual sensor that measures a distance between the image pickup means and a plurality of points 
on the object. Ninomiya discloses a robot vision system including a three-dimensional visual 
sensor whose image pickup means measures the distance between the image pickup means and a 
plurality of points on the object (column 4, line 28-49). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to utilize a three-dimensional visual 
sensor as taught by Ninomiya in order to determine the position and posture of an object without 
operation or accuracy being effected by contrast, color, or surface condition of the object 
(column 10, line 35-39). 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Ryan J. Hesseltine 
January 18,2005 



